aparler/PyNorch
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix backward reduce distributed

Browse source
This commit is contained in:
lucasdelimanogueira 2024-06-01 20:55:02 -03:00
parent 4b683a20e9
commit ef84c08841
2 changed files with 5 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -7,7 +7,7 @@ Project details explanations can also be found on [medium](https://towardsdatasc
**PyNorch** is a deep learning framework constructed using C/C++, CUDA and Python. This is a personal project with educational purpose only! `Norch` means **NOT** PyTorch, and we have **NO** claims to rivaling the already established PyTorch. The main objective of **PyNorch** was to give a brief understanding of how a deep learning framework works internally. It implements the Tensor object, GPU support and an automatic differentiation system.
# 2 - Installation
Install this package from PyPi (you can test on Colab! Also tested on AWS p3 instances ami-00f1d513c2bb78c75)
Install this package from PyPi (you can test on Colab!)
```css
$ pip install norch

5
norch/nn/parallel.py
View file

@ -13,8 +13,11 @@ class DistributedDataParallel(Module):
def backward(self):
self.module.backward()
for module, name, parameter in self.parameters():
for module, name, _ in self.parameters():
parameter = getattr(module, name)
dist.allreduce_mean_tensor(parameter)
setattr(module, name, parameter)